Home

home

buildLifecycle: experimental

Introduction

Xposome is a shiny application that aims to build a repository of gene expression datasets that profile the transcriptomic response to exposure of known toxins.


Local Installation

Must have R and Rstudio installed before proceeding:

To run Xposome application locally:

  • Download the source code from GitHub repo
  • Unzip the file and navigate to shinyApps folder
  • Open app.R within that folder
  • Click Run App button to launch the application in Rstudio

Dockerize Xposome

Not familiar with Docker? You can download it here and check out the R Docker tutorials.

To dockerize the Xposome application:

  • Download the source code from GitHub repo
  • Unzip the file and open the terminal. Use the cd command to navigate to where the dockerfile is stored. The dockerfile is a text file that contains all of the dependences and instructions of how to run the Xposome app.
  • Run the following command to build a docker image that contains the dependences of the Xposome app:
    docker build xposome .
    To check if the image is built succesfully,
    docker images
  • Create a docker container that contains the Xposome image and publish the application on the host machine. You can run on the console:
    docker run -d -p 3838:3838 \
      -v shinyApps/:/srv/shiny-server/ \
      -v shinyApps-log/:/var/log/shiny-server/ \
      xposome:latest
    -d is to run the container in detached mode
    -v is to mount the codebase or data files from the host directory onto the container directory <host_dir>:<container_dir>
    -p is to link the container port onto the host port and publish the application through the host domain <host_port>:<container_port>
    For more information about the syntax usage, see docker documentation

    To check if the container is built and running sucessfully,
    docker container ps
  • Once the app is up and running, you can visit the local host, http://localhost:3838 to vertify if the application is indeed hosted there.

The docker image for Xposome is also available to download at Docker Hub. To pull the image, just type the following command in the terminal:

docker pull montibot/Xposome

Additionally, check out how to configure http or https web posting with NGINX and Docker-Compose


Data Access

The Xposome application requires several structural datasets in order for it to run smoothly. We provide an API access to our data, see API Explorer.


Application Usage

There are two main parts of the Xposome application:

  • Portal Page
  • Moderator Page

The Portal Page

Contains a list of chemical screenings that exposed to high-throughput transcriptomics assays. Each screen provides information about the chemicals and cell-line that was used in the experiment. A detailed analysis was done to explore the gene expression set, gene set enrichment, and connectivity map of interested cell-line that exposed to a panel of known carcinogens.



Below is a snapshot of the Adipogenicity portal:



The Moderator Page

Requires access from authorized users. This restricted page only allows authorized users to make changes to the existed screens or to upload new screens to the portal page.

The default login credentials are set as:

  Username: Xposome
  Password: Xposome

Lastly, check out the documentation of how to reset password if users forgot their login credentials with Postfix

Title

Links

License

Developers

NGINX and Docker-Compose

NGINX & Docker-Compose

NGINX - An open source web server and reverse proxy technology used for hosting websites and applications

Docker-compose - A technology for enabling docker containers to communicate to each other

In this post, I will go over steps of how to create an unencrypt (http) or encrypt (https) traffic that allows users to interact with the Xposome application that is published through a docker container. To achieve that, we need both NGINX and docker-compose.

Here are the seven steps to set-up NGINX and docker-compose:

  1. Install docker-compose
  2. Build a NGINX image from Docker Hub without installing NGINX software
  3. Dockerize the Xposome application locally or pull the image from Docker Hub
  4. Configure NGINX configuration files
  5. Create a docker-compose.yml file
  6. Run the docker-compose.yml file
  7. Check to see if the application is running on http or https

Step 1: Install docker-compose

See how to install Docker-compose here

To check if docker-compose is installed, 

docker-compose --version

Step 2: Build a docker image for NGINX

An image for NGINX can be found at Docker Hub. You can pull this image without the need to install the NGINX software locally,

docker pull nginx:latest

To check if the image is built successfully, 

docker images

Also see this link on how to use the image

Step 3: Dockerize the Xposome application locally or pull the image from Docker Hub

Check out this post for more details

Step 4: Configure NGINX configuration files

There are two configuration files for NGINX, nginx.conf and default.conf.template files.

The nginx.conf file contains the standard configuration for NGINX. Unless you know how to add directives specified for a configuration file, otherwise, I do not recommend making any changes to this file.

The default.conf.templat file contains configuration that allows NGINX to direct any encrypt or unencrypt traffics to an application that is published on a specific port on the host machine.

How NGINX talks to Docker

With NGINX,
nginx → listens on → port 80 (http) and 443 (https) 

With Docker,
Docker → listens on → port 80/443/8080/3838/8000/… 

With SHINY
Shiny → listens on localhost → port 3838/8787/4848/…

In other words, when we run a docker container, we basically expose our application through a port that can be viewable thru a Shiny's localhost. Docker listens to this port and will publish the application to a specified port on the host machine (for example: port 7856 for simplicity). Therefore, when we navigate to port 7856 on the host domain (for example http://[domain_name/ip address]:7856), we will see that the shiny app is hosted there.

Nevertheless, we probably don't want to navigate to the application via a port link. Thus, we can use the NGINX configuration files to create an alias location for the application (for example http://[domain_name/ip address]/Xposome/). When a user is navigating to that specific address on the host domain, NGINX will transfer that traffic to the port 7856 which the application was originally published on. As the result, NGINX is served as a reverse proxy for hosting our applications.

Here is a snapshot of how the http configuration file looks like:

  server {
      listen       80;          
      server_name  155.41.202.164;  #ip address or domain name
      server_tokens off;

      # Load configuration files for the default server block.
      include /etc/nginx/default.d/*.conf;

      # Specific the location for the xposome app
      location /Xposome/ {
        rewrite ^/Xposome/(.*)$ /$1 break;
        proxy_pass http://example.com:7856/; 

        proxy_redirect off;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $server_name;

        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_read_timeout 20d;
        proxy_buffering off;
      }
  }

Step 5: Create a docker-compose.yml file

Once we had configured NGINX as a reverse proxy for our application, we can create a docker-compose.yml file that comprises of all the containers that we want to communicate with the NGINX container.

To enable communication between containers, in the docker-compose file, we must create a service to run the NGINX container and a service to run the container for the Xposome application. Under each service, we need to specify a list of instructions of how the container can be built, for instance, what image is used to build the container, what do we want to name the container, what port is used to expose the shiny app to Docker, and what port is used to publish the app onto the host machine.

After we had defined all of these keys components of how the containers are built, in the NGINX configuration files, we need to make sure that the host port matches the port that NGINX redirect the encrypt or unencrypt traffic to the application.

Here is a snapshot of the docker-compose.yml file:

version: '3'

services:
  nginx:
    image: nginx:latest         #name of the image
    container_name: nginx       #name of the container you want to build
    restart: always             #allow the connection to restart if it gets disconnected
    ports:
      - 80:80   #specify the unencrypt port
      - 443:443 #specify the encrypt port but will need ssl later
    volumes:
      - /home/docker/nginx/nginx.conf:/etc/nginx/nginx.conf #connect the host configuration files to docker config files
      - /home/docker/nginx/conf.d:/etc/nginx/templates #connect the host configuration files to docker config files
    command: [nginx-debug, '-g', 'daemon off;']

  xposome:
    image: montibot/xposome:latest  #name of the docker image
    container_name: xposome         #name of the container
    restart: always
    ports:
      - 7856:7856                   #port where the application is published on   
    volumes:
      - /home/xposome/shinyApps/:/srv/shiny-server/           #location to the codebse or data files
      - /home/xposome/shinyApps-log/:/var/log/shiny-server/   #location to store log information from the shiny app

Step 6: Run the docker-compose file

After we have the docker-compose file and NGINX configuration files all set up, we can cd to where docker-compose.yml is located and run the following command on the terminal.

docker-compose up -d

-d is used to run docker compose in detach mode

Step 7: Check to make sure the application is running on http or https

Navigate to the pre-specified domain (for example http://155.41.202.164/Xposome/) on host server and see if the application is hosted there.

Title

Links

License

Developers

Send Emails with Postfix

Send Emails with Postfix

Postfix - A popular open-source Mail Transfer Agent (MTA) that can be used to route and deliver emails on a Linux system

Docker - A software platform that allows developers to build, test, deploy, and run applications using containers

In this post, I will go over steps of how to send emails from a docker container through Postfix installed on the host machine. In my previous post, we have learned to dockerize the Xposome application. The Xposome application has two parts: the portal page and the moderator page. In the moderator page, we have an option that allow users to retrieve their passwords if they forgot their login credentials.

Here are five steps to set-up Postfix and send new passwords to users:

  1. Install Postfix
  2. Dockerize the Xposome application locally or pull the image from Docker Hub
  3. Configure Postfix to listen to requests from docker container
  4. Write an email to test the Postfix Mail Server
  5. Check to see if the email is sent correctly

Step 1: Install Postfix

Depending on your OS, for mine, I have CentOS 8 server. Therefore, I use this documenation on how to install and configure postfix mail server on CentOS 8. You can see this link to install and configure Postfix on Ubuntu 16.04.

Step 2: Dockerize the Xposome application locally or pull the image from Docker Hub

See my previous post here

Step 3: Configure Postfix to listen to requests from docker container

After Postfix is installed, the main configuration files is stored at /etc/postfix/main.cf.

To configure Postfix to listen to requests from docker container, there is something called docker bridge (docker0) which acts as a bridge between the ethernet port and the docker container so that data can go back and forth. Hence, we want to listen on the docker0 interface. To do that, type ifconfig on your host system to find out the bridge address and set your Postfix to listen on it.

As you can see in the image above, the IP address is 172.17.0.1

In the Postfix configuration file, set

inet_interfaces = 172.17.0.1

While you are there, add your actual docker container ip address to mynetworks

mynetworks = 172.17.0.5

172.17.0.5 is the private IP address of my docker container from which I use to send emails.

You can find the IP address of your docker container by

docker inspect [container_id/container_name]

For example,

docker inspect Xposome

Step 4: Write an email to test the Postfix Mail Server

Here is an email function that sends a temporary password to users who forgot their login credentials:

sendpassword <- function(from_sender="rchau88@bu.edu", to_recipient="lilychau999@gmail.com", recipient_first="Reina", recipient_last="Chau", recipient_account="Reina", tmp_pwd){

  recipient=paste(recipient_first, recipient_last)

  msg <- mime_part(
    paste0(
      '<!DOCTYPE>',
      '<html>',
      '<head>',
      '<meta http-equiv="Content-Type" content="text/html; charset=utf-8"/>',
      '<meta name="viewport" content="width=device-width, initial-scale=1.0"/>',
      '<title>HTML MESSAGE</title>',
      '<style type="text/css">',
      '</style>',
      '</head>',
      '<body>',
      '<p>Hi <strong>', recipient_first, ',</strong></p>',
      '<p>The password for your Xposome account has changed.</p>',
      '<p></p>',
      '<p>Username: <strong>', recipient_account, '</strong></p>',
      '<p>Temporary password: <strong>', tmp_pwd, '</strong></p>',
      '<br>',
      '<p>Best,</p>',
      '<p>Montilab Team</p>',
      '</body>',
      '</html>' 
    )
  )

  ## Override content type.
  msg[["headers"]][["Content-Type"]] <- "text/html"

  from <- paste0("\"Montilab Team\"<", from_sender, ">")
  to <- paste0("\"", recipient, "\"<", to_recipient, ">", collapse="")
  subject <- "Temporary password for Xposome"
  body <- list(msg)
  sendmail(from, to, subject, body, control=list(smtpServer="smtp.bu.edu", smtpPort="25"))

}

On the shiny interface, when a user clicks the forgot password? link, a modal dialog will pop up. After he/she provided the user account information and click the submit button, a new temporary password will be send to his/her email. 

observeEvent(input$FG_Button, {

  Firstname=trimws(input$FG_Firstname);
  Lastname=trimws(input$FG_Lastname);
  Username=trimws(input$FG_Username);

  login_dat <- read.csv(paste0("data/User_Login_List.csv"), header = TRUE, stringsAsFactors = FALSE)

  if(Firstname=="" | Lastname=="" | Username==""){

    forgotpasswordwarningmsg("Please fill in the required (*) fields.")

  }else{

    row <- which(login_dat$Username == Username)

    if(length(row) > 0){

      tmp_pwd <- password(n = 10, numbers = TRUE, case = TRUE, special = c("?", "!", "&", "%", "$"))
      login_dat$Password[row[1]] <- sodium::password_store(as.character(tmp_pwd))
      sendpassword(
        from_sender="rchau88@bu.edu",
        to_recipient="lilychau999@gmail.com", 
        recipient_first=Firstname, 
        recipient_last=Lastname, 
        recipient_account=Username, 
        tmp_pwd=tmp_pwd
      )
      write.csv(login_dat, paste0("data/User_Login_List.csv"), row.names = FALSE)
      forgotpasswordwarningmsg("Thank you for your submission! A temporary password has been sent to your email.")

    }else{

      forgotpasswordwarningmsg("This username does not exist in our database. Please enter another username.")

    }

  }
})

Here a snapshoot of the Xposome interface:

Step 5: Check to see if the email is sent correctly

if you are using the same email format that I have structured, then you should receive an email similar to this one.

Title

Links

License

Developers

API Explorer

API EXPLORER

The API Explorer provides documentation on how users can extract data from Xposome application and create their next project. All successful responses are returned in JSON. Only queries that respond with a 200 response code is considered successful. Here are standard HTTP codes you will find in the “Status” element of the response body.

Status Code Description
200 OK Standard HTTP successful response
400 Bad Request Standard HTTP invalid request response/td>
401 Unauthorized Standard HTTP unauthorized access response
500 Internal Server Error There was an unexpected error on our server. If you see this please contact montilab@bu.edu


Below are R code and packages that use to execute the API calls, but one can use python or other programs.


# R packages required for API calls
library(jsonlite)
library(httr)

RETRIEVE A LIST OF PORTALS AVAILABLE IN GENEHIVE

Property

Parameter Syntax Description
all Yes Whether to return all the portal names

API Structure

http://155.41.202.164:8989/portals?all=Yes

Example in R


# url for local testing
url <- "http://155.41.202.164:8989/portals?all=Yes"

# Send GET Request to API
res <- GET(url = url, encode = 'json')

stop_for_status(res)

portals <- fromJSON(fromJSON(rawToChar(res$content)))

print(portals)

portal_name = portals[2]

RETRIEVE A LIST OF CHEMICALS IN A PORTAL

Property

Parameter Syntax Description
portal MCF10A Name of the portal

API Structure

http://155.41.202.164:8989/chemicals?portal=MCF10A

Example in R


# url for local testing
url0 <- paste0("http://155.41.202.164:8989/chemicals?portal=", portal_name)

# Send GET Request to API
res <- GET(url = url0, encode = 'json')

stop_for_status(res)

chemicals <- fromJSON(fromJSON(rawToChar(res$content)))

print(head(chemicals))

chemical_id = chemicals[4]

RETRIEVE GENE EXPRESSION STATISTICS PER EACH CHEMICAL IN A PORTAL

Property

Parameter Syntax Description
portals MCF10A Name of the portal
chemical_id 1,2-Dibromo-3-chloropropane Name of the chemicals in the portal
summarize.func mean/median/min/max/Q1/Q3 Name of the summarize function
landmark TRUE or FALSE Whether to include landmark genes
do.nmarkers TRUE or FALSE Whether to filter the number of landmark genes
nmarkers 1 to 1000 Range of landmark genes
do.scorecutoff TRUE or FALSE Whether to filter the z-scores
scorecutoff -2 to 2 Range of z-score cutoff

API Structure

http://155.41.202.164:8989/gene_expression?portal=MCF10A&chemical_id=1,2-Dibromo-3-chloropropane&summarize.func=median&landmark=TRUE&do.markers=TRUE&do.scorecutoff=TRUE

Example in R


# url for local testing
url1 <- paste0("http://155.41.202.164:8989/gene_expression?portal=", portal_name, "&chemical_id=", chemical_id, "&summarize.func=median&landmark=TRUE&do.markers=TRUE&do.scorecutoff=TRUE")

# Send GET Request to API
res <- GET(url = url1, encode = 'json')

stop_for_status(res)

gene_expression_stat <- fromJSON(fromJSON(rawToChar(res$content)))

print(head(gene_expression_stat))

RETRIEVE GENE SET ENRICHMEMT STATISTICS PER EACH CHEMICAL IN A PORTAL

Property

Parameter Syntax Description
portals MCF10A Name of the portal
chemical_id 1,2-Dibromo-3-chloropropane Name of the chemicals in the portal
geneset Hallmark/C2/NURSA Name of the gene set enrichment
gsva gsva (default) The gene set enrichment method
summarize.func mean/median/min/max/Q1/Q3 Name of the summarize function

API Structure

http://155.41.202.164:8989/gs_enrichment?portal=MCF10A&chemical_id=1,2-Dibromo-3-chloropropane&geneset=Hallmark&gsva=gsva&summarize.func=median

Example in R


# url for local testing
url2 <- paste0("http://155.41.202.164:8989/gs_enrichment?portal=", portal_name, "&chemical_id=", chemical_id, "&geneset=Hallmark&gsva=gsva&summarize.func=median")

# Send GET Request to API
res <- GET(url = url2, encode = 'json')

stop_for_status(res)

gs_enrichment_stat <- fromJSON(fromJSON(rawToChar(res$content)))

print(head(gs_enrichment_stat))

RETRIEVE CONNECTIVITY STATISTICS PER EACH CHEMICAL IN A PORTAL

Property

Parameter Syntax Description
portals MCF10A Name of the portal
chemical_id 1,2-Dibromo-3-chloropropane Name of the chemicals in the portal
connectivity_classes pcl for Perturbagen Classes and pert for Perturbagens Name of the connectivity classes
summarize.func mean/median/min/max/Q1/Q3 Name of the summarize function

API Structure

http://155.41.202.164:8989/connectivity?portal=MCF10A&chemical_id=1,2-Dibromo-3-chloropropane&connectivity_classes=pcl&summarize.func=median

Example in R


# url for local testing
url3 <- paste0("http://155.41.202.164:8989/connectivity?portal=", portal_name, "&chemical_id=", chemical_id, "&connectivity_classes=pcl&summarize.func=median")

# Send GET Request to API
res <- GET(url = url3, encode = 'json')

stop_for_status(res)

connectivity_stat <- fromJSON(fromJSON(rawToChar(res$content)))

print(head(connectivity_stat))

RETRIEVE PROFILE ANNOTATION FOR A PORTAL

Property

Parameter Syntax Description
portal MCF10A Name of the portal

API Structure

http://155.41.202.164:8989/profile_annotation?portal=MCF10A

Example in R


# url for local testing
url4 <- paste0("http://155.41.202.164:8989/profile_annotation?portal=", portal_name)

# Send GET Request to API
res <- GET(url = url4, encode = 'json')

stop_for_status(res)

profile_annotation <- fromJSON(fromJSON(rawToChar(res$content)))

print(head(profile_annotation))

RETRIEVE CHEMICAL ANNOTATION FOR A PORTAL

Property

Parameter Syntax Description
portal MCF10A Name of the portal

API Structure

http://155.41.202.164:8989/chemical_annotation?portal=MCF10A

Example in R


# url for local testing
url5 <- paste0("http://155.41.202.164:8989/chemical_annotation?portal=", portal_name)

# Send GET Request to API
res <- GET(url = url5, encode = 'json')

stop_for_status(res)

chemical_annotation <- fromJSON(fromJSON(rawToChar(res$content)))

print(head(chemical_annotation))

RETRIEVE GENE EXPRESSION SET FOR A PORTAL

Property

Parameter Syntax Description
portal MCF10A Name of the portal

API Structure

http://155.41.202.164:8989/expression_set?portal=MCF10A

Example in R


# url for local testing
url6 <- paste0("http://155.41.202.164:8989/expression_set?portal=", portal_name)

# Send GET Request to API
res <- GET(url = url6, encode = 'json')

stop_for_status(res)

expression_set <- fromJSON(fromJSON(rawToChar(res$content)))

print(head(expression_set))

RETRIEVE GENE SET ENRICHMENT FOR A PORTAL

Property

Parameter Syntax Description
portal MCF10A Name of the portal
geneset Hallmark/C2/NURSA Name of the gene set enrichment
gsva gsva (default) The gene set enrichment method

API Structure

http://155.41.202.164:8989/enrichment_set?portal=MCF10A&geneset=Hallmark&gsva=gsva

Example in R


# url for local testing
url7 <- paste0("http://155.41.202.164:8989/enrichment_set?portal=", portal_name, "&geneset=Hallmark&gsva=gsva")

# Send GET Request to API
res <- GET(url = url7, encode = 'json')

stop_for_status(res)

enrichment_set <- fromJSON(fromJSON(rawToChar(res$content)))

print(head(enrichment_set))

RETRIEVE CONNECTIVITY MAP FOR A PORTAL

Property

Parameter Syntax Description
portal MCF10A Name of the portal
connectivity_classes pcl for Perturbagen Classes and pert for Perturbagens Name of the connectivity classes

API Structure

http://155.41.202.164:8989/connectivity_map?portal=MCF10A&connectivity_classes=pcl

Example in R


# url for local testing
url8 <- paste0("http://155.41.202.164:8989/connectivity_map?portal=", portal_name, "&connectivity_classes=pcl")

# Send GET Request to API
res <- GET(url = url8, encode = 'json')

stop_for_status(res)

connectivity_map <- fromJSON(fromJSON(rawToChar(res$content)))

print(head(connectivity_map))
Title

Links

License

Developers